We currently use dbsetup and dbupgrade for managing schema changes. A technical overview of these tools can be found here. This document discusses some of the challenges we face with our current tooling and proposes a migration to Liquibase. A couple of different migration strategies have been debated. Those strategies along with their pros and cons will be outlined as well. This document does not provide a thorough overview of dbsetup/dbupgrade or Liquibase. That information can be found elsewhere on the wiki or on the Liquibase site.
The most significant issue with dbsetup and dbupgrade is that we have to maintain and test two completely separate executions paths. This imposes a lot of overhead during a release cycle. With Liquibase however, everything is treated as an upgrade which means we have only a single execution path. This cuts our development and testing time in half. This single issue more than anything else is the motivation for doing the migration, and it cannot be stressed enough.
With dbsetup/dbupgrade changes are grouped into a schema spec. A schema spec has a unique numerical id. In the RHQ_SYSTEM_CONFIG table, we store the schema spec we are at under the key, DB_SCHEMA_VERSION. Let's say my development database is at 2.92, and the person who authored 2.89 realized something about it should have been different. Maybe a column should have been added or removed for example. The person makes the appropriate changes to the dbsetup and dbupgrade scripts, commits, and pushes the commit to the remote repo. All is well. The next time I git pull, I will get the updated scripts; however, since my schema is already at a version higher than 2.89, the changes will be ignored. And unless I am aware of the change, I may continue with my local development/testing for some period of time. This is problematic as it creates problems for development and testing. Liquibase solves this problem for us. In Liquibase, changes are grouped into a change set. An MD5 hash is recorded for each change set. If a change set that has already been applied to the database gets modified, a different MD5 hash will get generated. Liquibase will fail early when there are different MD5 sums for a change set that has already been applied, thereby bringing it to your attention sooner rather than later.
With dbsetup/upgrade, it is not possible to install/upgrade to a particular schema spec. Every spec that is declared (and that has not already run) is executed. Liquibase provides a lot of flexibility is this area. You can specify a change set, and everything not yet applied up to and include that change set will get executed. Liquibase also provides a form of tagging which you can apply to change sets. You can execute change sets having specific tags.
Two different strategies have been discussed. In the first approach, we make a clean break from dbsetup/dbupgrade such that all installations and upgrade are done completely through Liquibase. There are two major advantages with this. First, and most importantly, we only have to maintain and test a single execution path moving forward. Secondly, for the most part we only have to deal only with one tool. Having to juggle multiple tools/frameworks can be burdensome and adds to an already steep learning curve for newcomers. The main disadvantage with this approach is that it requires a substantial amount of upfront development and testing as all current scripts would have to be migrated. And we have to test all the various installation/upgrade paths that we support to ensure that we have not introduced regressions.
A second approach that has been discussed is to only use Liquibase post-RHQ 3.0.0. New installations would be done through Liquibase as would upgrades. In order to support doing new installations through Liquibase, we would have to migrate the dbsetup scripts. If a user is upgrading from 3.0.0 to 4.0.0 then the upgrade path would be solely through Liquibase as well. But if a user wants to migrate from a version earlier to 3.0.0 to 4.0.0, then we would have that user upgrade in two steps - first to 3.0.0 and then separately to 4.0.0. There are some advantages with this approach. This approach will result in significantly less development and testing to complete the migration than doing a 100% migration in which we make the clean break from dbsetup/dbupgrade. If a user is migrating from version prior to 3.0.0, he first upgrades to 3.0.0 which has already been thoroughly tested and is stable. For testing we don't have to worry about that initial upgrade. We only have to test going from 3.0.0 to 4.0.0. There are a couple downsides to this approach. First, we have to maintain (at least to some degree) both tools going forward. Secondly and more importantly, we still have to worry about testing separate install and upgrade paths, albeit not the to extent that we have today with dbsetup/dbupgrade.
jmarques: in the 2nd approach, if new installation are done through liquibase, that means that all of the schema.xml and data.xml files need to be converted to liquibase. excluding lines that only contain whitespace, let's look at some stats to ballpark this effort:
lines (largely irrelevant, just putting here for the sake of completeness as well as to show how they can be misleading)
schema.xml: 2181 lines in 18 files
data.xml: 380 lines in 7 files
upgrade.xml: 3215 lines in 1 file
structures (this is what will drive the effort as each structure has a corresponding mapped tag in the liquibase world)
schema.xml: 162 tables, 907 columns, (101 indexes and 65 constraints) on 266 fields total
data.xml: 169 data inserts into 15 tables (17 of which are parameterized with data captured from the installation UI)
upgrade.xml: 237 add columns, 226 alter columns, 36 create sequences, 40 delete columns, 206 direct sql, 3 drop sequence, 3 drop table (in 104 schemaSpecs / change sets)
so perhaps going the extra mile, since the 2nd strategy already has us converting the schema and data files, to also convert the upgrade file. the effort on dev isn't that much more, the real difference will be in the amount of automated testing we have around this stuff and the risk of regression. however, we're already introducing a large regression risk by creating an entire new liquibase-setup script, so we might as well go full throttle.